Time-of-flight (TOF) camera systems are known for providing information concerning the distance to an object by analysis of the Time of Flight from a light source to the object and back. One particular application is a camera capable of capturing a three-dimensional (3-D) image of a scene, that is, two-dimensional information, as well as, depth or distance information. Such camera systems are utilised in many applications where depth or distance information from a fixed point needs to be determined. Typically, the depth or distance information is measured from the TOF camera system.
The basic operational principle of a TOF camera system is to illuminate a scene with modulated light, such as pulses. The modulated light such as these light pulses are reflected back from objects within the scene and a lens collects the reflected light and forms an image of objects in the scene on an imaging sensor, and in particular, on a sensor plane of the sensor. Depending on the distance of objects from the camera, a delay is experienced between the emission of the modulated light, e.g. pulse and the reception of its reflection at the camera, for example, an object 2.5 m away from the camera causes a time delay of 16.66 ns. However, when using pulses, the pulse width of each light pulse determines the camera range, and, for example, for a pulse width of 50 ns, the range is limited to 7.5 m. As a consequence, the illumination of the scene becomes critical to the operation of a TOF camera system, and the requirements for illumination units necessitate the use of specialised light sources, such as, light emitting diodes (LEDs) or lasers, to generate such short light pulses.
Another main component of a TOF camera system is the imaging sensor. The imaging sensor typically comprises an array of pixels that form an image of the scene. In addition, the output of the pixels can be used to determine the time-of-flight of light from an illumination unit to an object in the scene and light reflected back to the imaging sensor from the object. The time-of-flight can be calculated in a separate processing unit which is coupled to the sensor or may be integrated into the sensor. Various methods are known for measuring the timing of the light as it travels from the illumination unit to the object and from the object back to the imaging sensor.
As is well-known, each pixel effectively comprises a photosensitive element which receives incident light and converts it into an electrical signal, for example, a current signal. In analogue timing imaging sensors, the pixels are connected to switches that direct the current signal to a memory acting as a summation device. In digital timing imaging sensors, high frequency counters are used for sensing the current signal. The presence of background light may cause distances to be erroneously determined as the pixels receive additional light not reflected from objects within the scene.
Naturally, imaging optics and processing electronics also form part of a TOF camera. The imaging optics gather the reflected light from objects in the scene and filter out light that is not in the same wavelength or frequency of the light emitted by the illumination unit. By filtering out light at unwanted wavelengths or frequencies, background light can effectively be suppressed. The processing electronics include drivers for both the illumination unit and the imaging sensor so that these components can accurately be controlled to ensure that a high resolution image of the scene can be obtained.
TOF camera systems tend to cover wide ranges from a few millimeters up to several kilometers depending on the type of imaging sensor utilised. Such a TOF camera system may have distance resolutions that vary from the sub-centimeter to several centimeters or even meters, depending on the configuration of the TOF camera system being used in a particular application. Technologies that can be used with TOF camera systems include pulsed light sources with digital time counters, radio frequency (RF) modulated light sources with phase detectors, and range-gated imagers.
Whilst it may be relatively easy to extract the distance information from the output of the imaging sensor for an object in a scene, TOF systems suffer from the impact of background light as discussed above; interference if more than one TOF camera system is operating with overlapping fields of view of the scene; and multiple reflections. Interference between different TOF camera systems can be reduced either by using time-multiplexing techniques or by utilising a different modulation frequency for each TOF camera system. As TOF camera systems illuminate an entire scene, multiple reflections may be obtained as the illuminating light reaches each object along several paths. This has the effect that the measured distance of the object from the TOF camera system may be determined as being greater than or less than its actual distance from the TOF camera system. This is a major drawback of conventional TOF camera systems. The removal of measurement ambiguity in range imaging systems has been described by A. D. Payne et al. in an article entitled “Multiple Frequency Range Imaging to Remove Measurement Ambiguity”, in which two different modulation frequencies are superimposed in a single capture to overcome the problem of phase ambiguity leading to incorrect distance measurements, (http://researchcommons.waikato.ac.nz/bitstram/10289/4032/1/Multiple %20Frequency %20Range%20Imaging.pdf).
In an article entitled “Resolving Depth Measurement Ambiguity with Commercially Available Range Imaging Cameras” by S. H. McClure et al., Proc. SPIE-IS&T Electronic Imaging, SPIE vol. 7538, pp. 75380K, 2010, software post-processing is used to resolve depth ambiguity. The range data is processed to segment the scene into separate objects and the average intensity of each object is used to determine which pixels are outside non-ambiguous range. The method can be used with any range imaging camera system, for example, robot vision where the camera may be moving during image captures, has reduced sensitivity to differences in reflectance from objects in the scene. In signal processing and related disciplines, aliasing refers to an effect that causes different signals to become indistinguishable (or aliases of one another) when sampled. Temporal aliasing is when the samples become indistinguishable in time. Temporal aliasing can occur when the signal being sampled periodically also has periodic content. In TOF systems aliasing results in ambiguity as to the distance between the light source and object from the might is reflected.
“Phase unwrapping” is described in an article entitled “Probabilistic Phase Unwrapping for Time-of-Flight Cameras” by David Droeschel, Dirk Holz and Sven Behnke, Proceedings of Joint 41st International Symposium on Robotics and 6th German Conference on Robotics, Munich, June 2010. A number of relative phase shifts are inferred from a wrapped phase signal where a probabilistic approach is used to detect phase jumps or phase shifts based on discontinuities in a depth image containing the distance information. The authors have also described phase unwrapping in “Multi-frequency Phase Unwrapping for Time-of-Flight Cameras”, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, October 2010.
In an article entitled “A New Active 3D-Vision System Based on RF-Modulation Interferometry of Incoherent Light” by R. Schwarte et al., Photonics East-Intelligent Systems and Advanced Manufacturing, Proceedings of the SPIE, Vol. 2588, Philadelphia, 1995, a method for processing signals from TOF camera system is disclosed in which an autocorrelation function utilises a phase shift algorithm where phase is proportional to distance from the TOF camera system and where measurement uncertainty can be determined in accordance with the number of phase measurements, the modulation contrast, the electrons in the active signal and in the noise signal and the wavelength of the modulation signal. Such systems are described as being useful for both pedestrian and vehicle safety systems.
In U.S. Pat. No. 7,791,715, a method of de-aliasing TOF phase-derived data by utilising at least two modulation frequencies to acquire phase data is described. Pixel detection information is captured in at least two discrete phases, preferably four, to implement de-aliasing. These discrete phases represent shifts between the modulator in the TOF camera system and the illumination unit. This is different to the desired phase shift detected between the emitted light and the reflected light from the object received by an imaging sensor. The discrete phases are subtracted from one another to cancel out fixed pattern offset between the modulator and the illumination unit. By using two modulation frequencies, the system behaves as if phase data is collected whilst it is being operated at a very slow modulation frequency which is proportional to the difference between two modulation frequencies. The use of multiple modulation frequencies provides better depth measurement certainty and precision than can be achieved using only one modulation frequency.
The use of multiple modulation frequencies is also described in “A Time-Of-Flight Depth Sensor—System Description, Issues and Solutions” by S. Burak Gokturk, Hakan Yalcin, and Cyrus Bamji of Canesta Inc. In one embodiment, two modulation frequencies are used where results are combined to provide a more accurate depth value for an object when compared to a single modulation frequency. In addition, the non-ambiguous or unambiguous range is extended. In another embodiment, a number of measurements are made with modulation frequencies at f, 2f, 4f, 8f etc., the results at each modulation frequency being combined to provide an accurate value of distance over a long non-ambiguous or unambiguous range can be obtained. In each successive measurement the resolution is doubled whilst the range is halved.
Due to the periodical nature of the illumination source used, namely, the modulation of the illumination source, distance range intervals tend to repeat. This phenomenon is known as TOF aliasing as described above. Currently, a limited number of techniques exist to solve this problem of TOF aliasing to enable the TOF camera system to provide a non-ambiguous or unambiguous distance of sufficiently long an interval for the particular application in which the TOF camera system is implemented. These techniques, however, have a number of known limitations as they tend to be quite complex to implement, require intensive computations, are not robust or cause bandwidth overheads in the data transport, or lead to a decrease of depth accuracy or general camera performance.